
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=Edge" />
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>tensorflowonspark.TFSparkNode module &#8212; TensorFlowOnSpark 1.3.3 documentation</title>
    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="tensorflowonspark.dfutil module" href="tensorflowonspark.dfutil.html" />
    <link rel="prev" title="tensorflowonspark.TFNode module" href="tensorflowonspark.TFNode.html" /> 
  </head><body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="tensorflowonspark.dfutil.html" title="tensorflowonspark.dfutil module"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="tensorflowonspark.TFNode.html" title="tensorflowonspark.TFNode module"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">TensorFlowOnSpark 1.3.3 documentation</a> &#187;</li>
          <li class="nav-item nav-item-1"><a href="tensorflowonspark.html" accesskey="U">tensorflowonspark package</a> &#187;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="module-tensorflowonspark.TFSparkNode">
<span id="tensorflowonspark-tfsparknode-module"></span><h1>tensorflowonspark.TFSparkNode module<a class="headerlink" href="#module-tensorflowonspark.TFSparkNode" title="Permalink to this headline">¶</a></h1>
<p>This module provides low-level functions for managing the TensorFlowOnSpark cluster.</p>
<dl class="class">
<dt id="tensorflowonspark.TFSparkNode.TFNodeContext">
<em class="property">class </em><code class="descname">TFNodeContext</code><span class="sig-paren">(</span><em>executor_id</em>, <em>job_name</em>, <em>task_index</em>, <em>cluster_spec</em>, <em>defaultFS</em>, <em>working_dir</em>, <em>mgr</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#TFNodeContext"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFNodeContext" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></p>
<p>Encapsulates unique metadata for a TensorFlowOnSpark node/executor and provides methods to interact with Spark and HDFS.</p>
<p>An instance of this object will be passed to the TensorFlow “main” function via the <cite>ctx</cite> argument.
To simply the end-user API, this class now mirrors the functions of the TFNode module.</p>
<dl class="docutils">
<dt>Args:</dt>
<dd><table class="first last docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">executor_id:</th><td class="field-body">integer identifier for this executor, per <code class="docutils literal notranslate"><span class="pre">nodeRDD</span> <span class="pre">=</span> <span class="pre">sc.parallelize(range(num_executors),</span> <span class="pre">num_executors).</span></code></td>
</tr>
<tr class="field-even field"><th class="field-name">job_name:</th><td class="field-body">TensorFlow job name (e.g. ‘ps’ or ‘worker’) of this TF node, per cluster_spec.</td>
</tr>
<tr class="field-odd field"><th class="field-name">task_index:</th><td class="field-body">integer rank per job_name, e.g. “worker:0”, “worker:1”, “ps:0”.</td>
</tr>
<tr class="field-even field"><th class="field-name">cluster_spec:</th><td class="field-body">dictionary for constructing a tf.train.ClusterSpec.</td>
</tr>
<tr class="field-odd field"><th class="field-name">defaultFS:</th><td class="field-body">string representation of default FileSystem, e.g. <code class="docutils literal notranslate"><span class="pre">file://</span></code> or <code class="docutils literal notranslate"><span class="pre">hdfs://&lt;namenode&gt;:8020/</span></code>.</td>
</tr>
<tr class="field-even field"><th class="field-name">working_dir:</th><td class="field-body">the current working directory for local filesystems, or YARN containers.</td>
</tr>
<tr class="field-odd field"><th class="field-name">mgr:</th><td class="field-body">TFManager instance for this Python worker.</td>
</tr>
</tbody>
</table>
</dd>
</dl>
<dl class="method">
<dt id="tensorflowonspark.TFSparkNode.TFNodeContext.absolute_path">
<code class="descname">absolute_path</code><span class="sig-paren">(</span><em>path</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#TFNodeContext.absolute_path"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFNodeContext.absolute_path" title="Permalink to this definition">¶</a></dt>
<dd><p>Convenience function to access <code class="docutils literal notranslate"><span class="pre">TFNode.hdfs_path</span></code> directly from this object instance.</p>
</dd></dl>

<dl class="method">
<dt id="tensorflowonspark.TFSparkNode.TFNodeContext.export_saved_model">
<code class="descname">export_saved_model</code><span class="sig-paren">(</span><em>sess</em>, <em>export_dir</em>, <em>tag_set</em>, <em>signatures</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#TFNodeContext.export_saved_model"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFNodeContext.export_saved_model" title="Permalink to this definition">¶</a></dt>
<dd><p>Convenience function to access <code class="docutils literal notranslate"><span class="pre">TFNode.export_saved_model</span></code> directly from this object instance.</p>
</dd></dl>

<dl class="method">
<dt id="tensorflowonspark.TFSparkNode.TFNodeContext.get_data_feed">
<code class="descname">get_data_feed</code><span class="sig-paren">(</span><em>train_mode=True</em>, <em>qname_in='input'</em>, <em>qname_out='output'</em>, <em>input_mapping=None</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#TFNodeContext.get_data_feed"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFNodeContext.get_data_feed" title="Permalink to this definition">¶</a></dt>
<dd><p>Convenience function to access <code class="docutils literal notranslate"><span class="pre">TFNode.DataFeed</span></code> directly from this object instance.</p>
</dd></dl>

<dl class="method">
<dt id="tensorflowonspark.TFSparkNode.TFNodeContext.start_cluster_server">
<code class="descname">start_cluster_server</code><span class="sig-paren">(</span><em>num_gpus=1</em>, <em>rdma=False</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#TFNodeContext.start_cluster_server"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFNodeContext.start_cluster_server" title="Permalink to this definition">¶</a></dt>
<dd><p>Convenience function to access <code class="docutils literal notranslate"><span class="pre">TFNode.start_cluster_server</span></code> directly from this object instance.</p>
</dd></dl>

</dd></dl>

<dl class="class">
<dt id="tensorflowonspark.TFSparkNode.TFSparkNode">
<em class="property">class </em><code class="descname">TFSparkNode</code><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#TFSparkNode"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFSparkNode" title="Permalink to this definition">¶</a></dt>
<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></p>
<p>Low-level functions used by the high-level TFCluster APIs to manage cluster state.</p>
<p><strong>This class is not intended for end-users (see TFNode for end-user APIs)</strong>.</p>
<p>For cluster management, this wraps the per-node cluster logic as Spark RDD mapPartitions functions, where the RDD is expected to be
a “nodeRDD” of the form: <code class="docutils literal notranslate"><span class="pre">nodeRDD</span> <span class="pre">=</span> <span class="pre">sc.parallelize(range(num_executors),</span> <span class="pre">num_executors)</span></code>.</p>
<p>For data feeding, this wraps the feeding logic as Spark RDD mapPartitions functions on a standard “dataRDD”.</p>
<p>This also manages a reference to the TFManager “singleton” per executor.  Since Spark can spawn more than one python-worker
per executor, this will reconnect to the “singleton” instance as needed.</p>
<dl class="attribute">
<dt id="tensorflowonspark.TFSparkNode.TFSparkNode.cluster_id">
<code class="descname">cluster_id</code><em class="property"> = None</em><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFSparkNode.cluster_id" title="Permalink to this definition">¶</a></dt>
<dd><p>Unique ID for a given TensorFlowOnSpark cluster, used for invalidating state for new clusters.</p>
</dd></dl>

<dl class="attribute">
<dt id="tensorflowonspark.TFSparkNode.TFSparkNode.mgr">
<code class="descname">mgr</code><em class="property"> = None</em><a class="headerlink" href="#tensorflowonspark.TFSparkNode.TFSparkNode.mgr" title="Permalink to this definition">¶</a></dt>
<dd><p>TFManager instance</p>
</dd></dl>

</dd></dl>

<dl class="function">
<dt id="tensorflowonspark.TFSparkNode.inference">
<code class="descname">inference</code><span class="sig-paren">(</span><em>cluster_info</em>, <em>feed_timeout=600</em>, <em>qname='input'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#inference"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.inference" title="Permalink to this definition">¶</a></dt>
<dd><p>Feeds Spark partitions into the shared multiprocessing.Queue and returns inference results.</p>
<dl class="docutils">
<dt>Args:</dt>
<dd><table class="first last docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">cluster_info:</th><td class="field-body">node reservation information for the cluster (e.g. host, executor_id, pid, ports, etc)</td>
</tr>
<tr class="field-even field"><th class="field-name">feed_timeout:</th><td class="field-body">number of seconds after which data feeding times out (600 sec default)</td>
</tr>
<tr class="field-odd field"><th class="field-name">qname:</th><td class="field-body"><em>INTERNAL_USE</em></td>
</tr>
</tbody>
</table>
</dd>
<dt>Returns:</dt>
<dd>A dataRDD.mapPartitions() function</dd>
</dl>
</dd></dl>

<dl class="function">
<dt id="tensorflowonspark.TFSparkNode.run">
<code class="descname">run</code><span class="sig-paren">(</span><em>fn</em>, <em>tf_args</em>, <em>cluster_meta</em>, <em>tensorboard</em>, <em>log_dir</em>, <em>queues</em>, <em>background</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#run"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.run" title="Permalink to this definition">¶</a></dt>
<dd><p>Wraps the user-provided TensorFlow main function in a Spark mapPartitions function.</p>
<dl class="docutils">
<dt>Args:</dt>
<dd><table class="first last docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">fn:</th><td class="field-body">TensorFlow “main” function provided by the user.</td>
</tr>
<tr class="field-even field"><th class="field-name">tf_args:</th><td class="field-body"><code class="docutils literal notranslate"><span class="pre">argparse</span></code> args, or command line <code class="docutils literal notranslate"><span class="pre">ARGV</span></code>.  These will be passed to the <code class="docutils literal notranslate"><span class="pre">fn</span></code>.</td>
</tr>
<tr class="field-odd field"><th class="field-name">cluster_meta:</th><td class="field-body">dictionary of cluster metadata (e.g. cluster_id, reservation.Server address, etc).</td>
</tr>
<tr class="field-even field"><th class="field-name">tensorboard:</th><td class="field-body">boolean indicating if the chief worker should spawn a Tensorboard server.</td>
</tr>
<tr class="field-odd field"><th class="field-name">log_dir:</th><td class="field-body">directory to save tensorboard event logs.  If None, defaults to a fixed path on local filesystem.</td>
</tr>
<tr class="field-even field"><th class="field-name">queues:</th><td class="field-body"><em>INTERNAL_USE</em></td>
</tr>
<tr class="field-odd field"><th class="field-name">background:</th><td class="field-body">boolean indicating if the TensorFlow “main” function should be run in a background process.</td>
</tr>
</tbody>
</table>
</dd>
<dt>Returns:</dt>
<dd>A nodeRDD.mapPartitions() function.</dd>
</dl>
</dd></dl>

<dl class="function">
<dt id="tensorflowonspark.TFSparkNode.shutdown">
<code class="descname">shutdown</code><span class="sig-paren">(</span><em>cluster_info, queues=['input']</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#shutdown"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.shutdown" title="Permalink to this definition">¶</a></dt>
<dd><p>Stops all TensorFlow nodes by feeding <code class="docutils literal notranslate"><span class="pre">None</span></code> into the multiprocessing.Queues.</p>
<dl class="docutils">
<dt>Args:</dt>
<dd><table class="first last docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">cluster_info:</th><td class="field-body">node reservation information for the cluster (e.g. host, executor_id, pid, ports, etc).</td>
</tr>
<tr class="field-even field"><th class="field-name">queues:</th><td class="field-body"><em>INTERNAL_USE</em></td>
</tr>
</tbody>
</table>
</dd>
<dt>Returns:</dt>
<dd>A nodeRDD.mapPartitions() function</dd>
</dl>
</dd></dl>

<dl class="function">
<dt id="tensorflowonspark.TFSparkNode.train">
<code class="descname">train</code><span class="sig-paren">(</span><em>cluster_info</em>, <em>cluster_meta</em>, <em>feed_timeout=600</em>, <em>qname='input'</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/tensorflowonspark/TFSparkNode.html#train"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#tensorflowonspark.TFSparkNode.train" title="Permalink to this definition">¶</a></dt>
<dd><p>Feeds Spark partitions into the shared multiprocessing.Queue.</p>
<dl class="docutils">
<dt>Args:</dt>
<dd><table class="first last docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">cluster_info:</th><td class="field-body">node reservation information for the cluster (e.g. host, executor_id, pid, ports, etc)</td>
</tr>
<tr class="field-even field"><th class="field-name">cluster_meta:</th><td class="field-body">dictionary of cluster metadata (e.g. cluster_id, reservation.Server address, etc)</td>
</tr>
<tr class="field-odd field"><th class="field-name">feed_timeout:</th><td class="field-body">number of seconds after which data feeding times out (600 sec default)</td>
</tr>
<tr class="field-even field"><th class="field-name">qname:</th><td class="field-body"><em>INTERNAL_USE</em></td>
</tr>
</tbody>
</table>
</dd>
<dt>Returns:</dt>
<dd>A dataRDD.mapPartitions() function</dd>
</dl>
</dd></dl>

</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h4>Previous topic</h4>
  <p class="topless"><a href="tensorflowonspark.TFNode.html"
                        title="previous chapter">tensorflowonspark.TFNode module</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="tensorflowonspark.dfutil.html"
                        title="next chapter">tensorflowonspark.dfutil module</a></p>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="_sources/tensorflowonspark.TFSparkNode.rst.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <div class="searchformwrapper">
    <form class="search" action="search.html" method="get">
      <input type="text" name="q" />
      <input type="submit" value="Go" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    </div>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="tensorflowonspark.dfutil.html" title="tensorflowonspark.dfutil module"
             >next</a> |</li>
        <li class="right" >
          <a href="tensorflowonspark.TFNode.html" title="tensorflowonspark.TFNode module"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">TensorFlowOnSpark 1.3.3 documentation</a> &#187;</li>
          <li class="nav-item nav-item-1"><a href="tensorflowonspark.html" >tensorflowonspark package</a> &#187;</li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &#169; Copyright 2018, Yahoo Inc.
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.7.9.
    </div>
  </body>
</html>